1. Field of the Invention
The present invention relates generally to image processing, and more particularly to motion compensation processing for an image sensor that has a block-based analog-to-digital converter architecture.
2. Description of the Related Art
Digital image capturing devices use image sensors to convert incident light energy into electrical signals. Image sensor designs include Charged Coupled Devices (CCD), Complementary Metal Oxide Silicon (CMOS) image sensors, and Digital Pixel System (DPS) sensors. An image sensor includes a two-dimensional array of light sensing elements called pixels. Each pixel in the array works with the lens system to respond to incident light within a local area of the scene, and produces an electrical signal describing the characteristics of the scene. The electrical signals that are output from the light sensing elements are converted to digital form, and the resulting digital pixel values form the raw data representing the scene. The raw data can be processed by an image processor to produce rendered digital images.
Image sensors can be designed to support either rolling shutter or global shutter exposure methods. The difference between rolling shutter and global shutter resides in the timing that the pixels in the array are reset and exposed to light. Specifically, rolling shutter refers to a method wherein the pixels in the sensor are reset and exposed to light one group at a time (e.g., one row at a time), thus resulting in a delay in reset and exposed time between consecutive groups. Global shutter refers to a method wherein all the pixels in the array are reset and exposed to light at substantially the same time. U.S. Pat. No. 8,953,075 titled “CMOS image sensors implementing full frame correlated double sampling with global shutter”, the contents of which are incorporated herein by reference in their entirety, describes CMOS image sensor examples with global shutter and full frame correlated double sampling support.
The ability of an imaging device to resolve fine details is often referred to as the resolution of the imaging device. One measure of resolution includes an area resolution, which corresponds to the total number of pixels in the image sensor (or the number of pixels in the image sensor that contribute to an image, i.e. number of pixels per image frame). Another measure of resolution includes a pixel-density resolution, which corresponds to the number of pixels per predetermined area in the imaging device with a given lens focal length. For imaging application that utilize multiple images across time, such as capturing video, another measure of resolution includes time resolution, which may correspond to the number of captured image frames per second. The area resolution reflects the ability of the system to resolve details in the scene, whereas the time resolution reflects the ability of the system to resolve motions or changes in the scene.
As noted above, image sensors generally convert the analog signals that are output from the pixels into digital form. Image sensors typically include analog-to-digital convertors (ADCs) to perform such conversion, and various readout and A/D conversion architectures exist. Because pixel readout and A/D conversion take time and because there are numerous pixels in modern image sensors, the time it takes to readout out a single image frame (and hence the time resolution of the device) depends significantly on the pixel readout and A/D conversion architecture. In order to increase the time resolution of the image sensor, most CMOS or DPS image sensor designs include multiple ADCs and readout circuits to allow for faster readout and A/D conversion processing. For example, in a rectangular image sensor of M pixels high and N pixels wide, in general the maximum number of frames per second F would be given by F=(K*D)/(M*N), where K is the number of A/D conversions each ADC can perform per second and D is the number of ADCs in the image sensor. Thus, increasing the number of ADCs increases the maximum frame rate F.
One exemplary readout and A/D conversion architecture is a column-ADC architecture, in which each column of pixels in the image sensor shares a single ADC. In such an architecture, the maximum number of frames per second F would be given by F=K/M. A modified column-ADC architecture may include more than one ADC per column to increase the resolution of the image sensor, in which case the maximum number of frames per second F would be given by F=(K*L)/M, where L is the number of ADCs per column.
Another exemplary readout and A/D conversion architecture is a Digital Pixel System (DPS) sensor, in which ADCs are included in the circuits of the pixels themselves. For example, an ADC may be included in the circuit of each pixel or in the circuit of one pixel out of each group of pixels, where each group includes a small number of pixels (for example, four pixels). In this architecture, the maximum number of frames per second F would be given by F=K/G, where G is the number of pixels per group sharing an ADC. U.S. Pat. No. 6,975,355, the contents of which are incorporated herein by reference in their entirety, describes an exemplary DPS image sensor. This architecture has a very high throughput, and is particularly suitable for global shutter operation and for providing excellent wide dynamic range performance. However, potential disadvantages of the DPS architecture are that the circuit complexity is relatively high, and the method to schedule the operating events in the sensor can be relatively complicated.
Another exemplary readout and A/D conversion architecture is a block-ADC architecture (also known as an area-ADC architecture). In an image sensor with a block ADC design, the pixel array in the image sensor is partitioned into blocks (areas)—for example rectangular blocks of P pixels high by Q pixels wide. The pixels in each block (area) share a single ADC. Hence an image sensor of M pixels high by N pixel wide will use a total number of ADCs equal to (M/P)*(N/Q), assuming that M and N are integer multiples of P and Q, respectively. In the case either M/P or N/Q or both is not an integer, the quantities are rounded up to the next higher integer. In this architecture, the maximum number of frames per second F would be given by F=(K)/(P*Q). An image sensor design with a block ADC has various advantages, including very high throughput, the ability to provide either global shutter or block rolling shutter operation modes, and the ability to be implemented very compactly in silicon using a multi-layered stacked design.
An exemplary image sensor having a block-ADC architecture may be found in the paper “A block-parallel signal processing system for CMOS image sensor with three-dimensional structure,” by Kiyoyama et al., published in 3D Systems Integration Conference, pp. 1-4, 2010. Kiyoyama suggests an image sensor having a block-ADC architecture in which one-hundred pixels within a rectangular region share a single ADC and correlated double sampling (CDS) circuits. The circuits of the image sensor are designed in a stacked structure where the sensor elements sit on the top layer. The lower layers include the CDS and the ADC circuits. Sensor elements and circuits in the layers are connected by through holes (through silicon vias, or TSVs) in the integrated circuit. US Publication No. 2013/0236048, the contents of which are incorporated herein by reference in their entirety, discloses an embodiment similar to the stack design of Kiyoyama et al. The device of US2013/023 6048 integrates an image processor within the sensor to implement certain image processing operations such as gradient and key-point detection. The paper “A low-noise high-frame-rate 1 D-decoding readout architecture for stacked image sensors,” by Khakoni et al, published in IEEE Sensors Journal, pp. 1966-1973, February 2014, discusses a similar stacked architecture where a rectangle region of pixels shares a single ADC. The design of Khakoni et al. uses a sigma-delta ADC, and it incorporates a 1-D decoding structure to reduce the complexity of the circuitry.
In the block-ADC architecture, the ADCs are often provided in a separate layer from the pixel circuits according to a 3D or stacked image sensor design (such as in the image sensor of Kiyoyama et al.). However, this is not required. In particular, the block-ADC architecture in general does not specify the location of the ADC. Thus, the DPS architecture may be thought of as a special case of the block-ADC architecture, since each group of pixels in the DPS sensor that shares an ADC may be considered a block according to the block-ADC architecture.
FIG. 1 is a block diagram of an exemplary imaging system 10 using an image sensor 100 with the block-ADC architecture. The imaging system 10 includes an optical device 101, the image sensor 100, and an image signal processor 110. The optical device 101 includes one or more lenses that focus incident light onto the image sensor 100. The image sensor 100 includes an array 102 of light sensing elements (pixels), block-ADC circuitry 103, readout circuitry 104, and sequencer/timing control circuitry 108. The light sensing elements of the array 102 and the block-ADC circuitry 103 can be fabricated on different layers and connected via TSVs. The readout circuitry 104 interfaces the output of the block-ADC circuitry 103 and delivers the pixel data to the output of the sensor 100. The sequencer/timing control circuitry 108 controls the ordering and the timing for reading the pixels in the array 102.
As shown in FIG. 2, the pixels of the image sensor 100 are grouped into blocks 200 of P*Q pixels 201. The block-ADC circuitry 103 includes one ADC (not illustrated) per block 200 of P*Q pixels, such that the signals output from each pixel 201 in a given block 200 are A/D converted by the corresponding ADC of the block-ADC circuitry 103. Various exposure/readout methods can be designed for the P*Q pixels 201 in each block 200.
In a block global shutter exposure/readout method, all the pixels 210 in the array 102 are exposed at the same time and thereafter the pixel values in each block 200 are read out sequentially through the ADC corresponding to the block 200, for example in the order 0, 1, . . . PQ-1 that is shown in FIG. 2. That is, after all of the pixels 201 in the entire array 102 have been exposed, the all of 0th pixels in each block 200 of the array 102 is read out simultaneously (through their respectively corresponding ADC), followed by all the 1st pixels in each block 200 of the array 102 being read out simultaneously (through their respectively corresponding ADC), and so on. After exposure, the pixel values of the block 200 are temporarily stored (for example within the respective pixels 201) while awaiting their turn to be read out of the sensor 100. When the global shutter exposure method is not used, an image of a fast moving object may be distorted because the pixels 201 are not exposed at the same time and the object may have moved noticeably between the time when a first pixel is exposed and the time when a last pixel is exposed. However, when the global shutter exposure method is used this distortion is reduced because all pixels 201 are exposed at the same time.
In a block rolling shutter readout operation, the pixels 201 in the array 102 are not all exposed at the same timing, but rather are exposed sequentially in groups, with read out for each pixel 201 following immediately after the exposure for that pixel ends. For example, each pixel 201 in a block 200 may be exposed and read out at a different timing from the other pixels 201 in the same block 200, and corresponding pixels 201 in different blocks 200 may be exposed and read out at the same timings. For example, all the 0th pixels in each block 200 of the array 102 are exposed and read out together, followed by all the 1st pixels in each block 200 of the array 102 being exposed and read out together, and so on up to the (PQ−1)th pixels 201. The ordering of the pixel readout shown in FIG. 2 follows a raster scan strategy within each block, i.e. scan the rows from the top to the bottom of each block, and for each row from the left to the right.
FIG. 3 shows the timing of the block rolling shutter readout scheme. Each phase corresponds to the pixels of a sequence number shown in FIG. 2, and since there are P*Q pixels 201 in a block 200, there are P*Q timing phases. Each rectangle with length E represents an exposure and A/D conversion time period for the corresponding phase having a duration time of E seconds. Each phase can begin exposure before the previous phase is completely finished. Since the pixel data are to be A/D converted, the timing of the rolling shutter approach is appropriate as long as the delay time D between each phase is sufficient to ensure that A/D conversion of the preceding phase is completed before A/D conversion of the next phase begins. The time required to scan through all phases is P*Q*D seconds, and hence the maximum possible frame rate is 1/(P*Q*D) frames per second. In practice, there are vertical and horizontal blanking times and hence the frame rate is lower than 1/(P*Q*D). Instead of the raster scanning method, a pseudo random ordering can also be used where the sequence numbers within each block of P by Q pixels are assigned in a pseudo random ordering.
FIG. 4 shows a readout scheme which can be referred to as partial global shutter (PGS), and is especially suitable for color image sensor designs. FIG. 4 (a) shows that the block size of the block ADC is two by two pixels, i.e., P=Q=2. For this design, there are only four phases which are designated according to a 2×2 pixel area. In the PGS design, a block rolling shutter readout operation is used, such as that discussed above. However, because the number of pixels 201 per block 200 in the PGS design is relatively low (e.g., four pixels 201 per block 200), an effect of the readout operation is similar to an effect of a global shutter readout operation. In particular, because only a small number of phases are required, a time between an exposure of a first pixel 201 and a time between an exposure of the last pixel 201 is small, and thus an opportunity for distortion is lowered. Moreover, if each block 200 corresponds to a quad unit of 4 pixels of a color image sensor (discussed further below), then each unit pixel in the ultimate rendered image will have been exposed essentially simultaneously as one another, and thus the effect of the global shutter will be approximated in the ultimate image. In some embodiments, a Bayer color pattern may be used, and each block 200 may correspond to a 2×2 unit of the Bayer pattern as shown in FIG. 4(b). In such a case, each phase corresponds to a color in the Bayer pattern, e.g.: phase 0 corresponds to R, phase 1 corresponds G1 (or Gr), phase 2 corresponds to G2 (or Gb), and phase 3 corresponds to B. The timing of the pixel exposure is illustrated in FIG. 5. This arrangement has an advantage that it can be operated at very high frame rate since there are only four phases. However, the small block size of 2×2 also means that the design requires a relatively large quantity of circuit components to implement the ADCs. It turns out that the PGS design in FIG. 4 also has smaller motion artifact compared to designs with larger block sizes when the sensor 100 moves during image capture. Of course, the PGS design can be used with more than four phases, e.g. 8 phases using a 2×4 or 4×2 block 200 size, 16 phases using a 4×4 block 200 size, . . . , etc.
It is also possible to design an image sensor with a block size of P by Q pixels where P*Q>X, where X is the number of phases (i.e., the number of phases per frame period is less than the number of pixels per block). For example, FIG. 6 illustrates an image sensor design in which each block has 16 pixels but with only four exposure/readout phases. In this case, the pixel values for each P*Q block will need to be stored and read out sequentially within a frame period similar to the design in a full global shutter sensor 100.
When an image sensor 100 is used in a portable electronic device 10 such as a camera, the image sensor 100 is frequently subjected to motion due to movement of the camera holder (whether a human being or a mechanical device). Such motion causes geometric distortions in the image when the image sensor 1000 is not a global shutter design, i.e., when the pixels are not all exposed at the same time. Motion artifacts are especially observable when the camera is operating under zoom conditions, e.g., with a zoom lens. Modern digital cameras can support relatively high optical zoom factors such as 50X. For a fixed range of motion and a fixed shutter speed, a larger zoom factor will result in a higher distortion since the effect due to motion is magnified by the optical zoom system. The artifacts due to motion can cause the images taken by the camera to be highly objectionable. This can mean that a picture taken with a hand-held camera even with a relatively steady hand can have unacceptable motion distortions especially when operating at a high zoom condition.
Accordingly, motion compensation, which uses motion information to correct for motion shifts, has been developed in the video compression field. Such motion compensation methods typically work in the YCbCr color space, and frequently in sub-sampled arrangements such as 4:2:2. In other words, the known motion compensation methods in video compression work on image data that has already been processed by an image signal processor.